Ar uuauo / h-o^. 



Time and Space Multiplexing Focal Plane Convolvers 

Paul E. Beaudet 

Westinghouse Electric Corporation, 
Advanced Technology Division 
P.O. Box 1521, MS 3D12 
Baltimore, Maryland 21203 
(301) 765-3752, Fax (301) 765-7652 



ABSTRACT 

Two varieties of Focal Plane Convolvers have been fabricated; the time multiplexing 
and the space multiplexing focal plane convolvers. They extract spacial features of a 
scene using parallel processing procedures at the imaging detector array site. The de- 
tailed description and operation of these devices is described. The following related 
areas of technology are also discussed: 

o Double Correlated Double Sampling o CCD Charge Transport 

o Equal Variance Mapping o Charge Transport Efficiency 

o Spacial Convolution Operators o Uniformity Issues 

o Local Rotational Invar iance o Other Devices on the Reticle 

1 . INTRODUCTION 

The Focal Plane Convolver (FPC) program* supported the development of two types of 
analog image processing chips; a Time Multiplexing Focal Plane Convolver (TMFPC) and a 
Space Multiplexing Focal Plane Convolver (SMFPC) . The TMFPC performs parallel processing 
by organizing packets of photo-converted charges over different time intervals while the 
SMFPC does its processing by organizing packets of photo-converted charges at different 
spacial locations over the same time interval. Both varieties were designed, fabricated, 
and tested. 

These convolvers are capable of extracting linear spacial features as signatures to 
indicate the presence of a target. Both chips perform, in parallel, local "match filter" 
processing at the detector site(s)* Three varieties of FPC chips were fabricated as well 
as many test structures: 1) An 11x11 TMFPC, 2) a 22x22 TMFPC and 3) a 120x120 SMFPC. 

Three wafer lots were processed and tested. The test results are encouraging showing 
.99998 charge transfer efficiency, greater than 5% yield on 11x11 TMFPC arrays (excluding 
design flaws) , charge handling capacity greater than 3000 electrons/// 2 and we successfully 
demonstrated TMFPC readout. Unfortunately, the results for lot 3 (for which many design 
flaws were corrected) still had high resistivity shorts between the two levels of 
polys ilicon to the extent that imaging could not be demonstrated. The test results are 
sufficiently promising however, that additional funding is being sought to continue the 
effort. * 

In section 2, the TMFPC is described; it is shown how this chip performs spacial 
convolutions. In section 3, the SMFPC is described similarly. Section 4 contains a de- 
scription of several related technologies including equal variance mapping and other de- 
vices on the reticle. Section 5, gives a summary and concludes with our current views of 
FPC technology. 

2. TIME MULTIPLEXING FOCAL PLANE CONVOLVER 

A focal plane convolver is a solid state electro-optical device capable of linearly 
combining opto-electronic signals from neighboring pixel sites for the purpose of ex- 
tracting spacial or temporal signatures (convolutions) in images formed on the focal plane 
of a camera. The TMFPC combines signals integrated at neighboring sites over different 
time intervals (time multiplexing) . 

The output of a convolution process, (e.g., matched filtering) is a linear weighting 
of neighborhood (i,j) picture elements (pixels) with positive or negative weights, a ii . 
These weights are referred to as the convolution kernel . Entities familiar to you such as 
the x-derivative, I x , or y-derivative, I y , of a scene are examples of convolutions. The 
Laplacian operator d 2 1 d 2 1 

+ denoted by I xx +I yy (derivatives are denoted by 

dx 2 . dy 2 

*This work was funded by DARPA and administered fry ARDC under contract No. D A AA2 1 - £■ 5 - C - 0 2 9 6 
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subscripts) is another example of a convolution. It is used to enhance the spacial fre- 
quencies of a scene. There are many philosophies for convolution kernels. Different sets 
of convolution kernels are desired as pre-processing operations for image understanding. 
To detect edges and corners in an image could typically require five separate convolutions; 
two for edges l x , I , and three for detecting corners I iy , I yy . Examples of 4x4 

kernels for each of these is shown below: 
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When the TMFPC creates a convolution by combining charge in the neighborhood of each 
pixel, another image in generated. In a typical convolution, C ijf some of the coefficients 
a ln are negative, the others are positive: 



= 2 
1,» 



Charge coupled devices (CCD) permit the accumulation of charge (electrons) in local buck- 
ets. When light falls on an unshielded depleted silicon region, the charge generation rate 
is proportional to the intensity of light falling on that region. By moving a (light 
shielded) charge bucket to different neighboring optically active locations, collecting 
charge there, and varying the dwell time at each region, a linear combination of neigh- 
boring pixel photo-converted charge is accumulated in the bucket. See Figure 1. But only 
positive coefficients (contributions) are permitted in such a process. To accommodate 
negative coefficients, two buckets must be utilized and the difference taken of the accumu- 
lated charge in both buckets. A light shielded area must be provided so that both the 
negative and positive contributions can "hide" while accumulating these contributions. 
We use a two dimensional CCD light shielded "roadway" system which accumulates, in parallel, 
the convolution contributions for every output pixel. The bucket pairs are clocked in the 
east-west roadway to reach neighboring pixels columns while already processed convolutions 
are simultaneously read out north to south. The difference in the positive and negative 
contributions to the convolution is taken using double correlated double sampling (DCDS) 
readout circuits. At "each (1, m) neighbor, light is accumulated an amount of time propor- 
tional to the convolution coefficient magnitude 1^ m |. This performs the equivalence of 
multiplying the intensity by [oc 1 n |. If the a Kernel coefficient ^ _ m is positive (nega- 
tive) the integrated photo-converted charge is transferred to the positive (negative) CCD 
bucket. When all l,m" contributions have been accumulated for a given convolution, the 
bucket pairs are placed in the north-south highways for readout while the now vacant east- 
west buckets can be used for the next convolution; the readout process and next convolution 
process can occur simultaneously. Arbitrary convolution are possible limited only by chip 
timing considerations, the amount of light available, and the charge handling capacity of 
the buckets. A photograph of the TMFPC imaging chip is shown in Figure 2. 

3. SPACE MULTIPLEXING FOCAL PLANE CONVOLVER 

The Space Multiplexing Focal Plane Convolver works by "slicing" the optical image into 
sub-pixel portions and distributing them to many local charge bucket pairs for the execu- 
tion of the desired convolution integral. Because each convolution is a local grouping of 
several sub-pixel charges, the convolutions are limited to those that have kernels with 
small integer coefficients. 

In the basic concept of a SMFPC, one considers the image pixel as a loqical pixel 
which is physically split into sub-pixels by the imaging focal plane. Today, it is possi- 
ble to create 5 M x Su sub-pixels and soon 3? x 3u sub-pixel technology will be available. 
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The sub-pixels can be smaller than the Airy disc (blur circle) of the optical focussing 
system. The proto-type SMFPC uses 20 P sub-pixel resolution as shown in Figure 3. The 
SMFPC design has 120x120 sub-pixels, but much larger arrays of much smaller sub-pixels are 
feasible. 

The basic concept of the SMFPC is illustrated in Figure 4" where each logical pixel is 
divided into four sub-pixels and the I xy convolution is performed by combining two sub- 
pixels from adjacent logical pixel corners into positive and negative contributing charge 
buckets. As can be seen, the space is multiplexed so that charge is combined into bucket 
pairs for each convolution (logical output pixel); there are as many convolutions (bucket 
pair read-outs) as there are logical pixels . 

The motion needed of sub-pixel charges for aggregation into their appropriate P + and 
P. output charge buckets is controlled by feeding each image line of sub-pixel charge into 
a multiplexing region (see Figure 5) segmented in time so that the order in which charges 
from each line enter the multiplexer is controlled (module 12) . For example, it is possi- 
ble that all sub-pixel charges j , having j -modulo 12=7 , "fall" into the multiplexer first 
followed then by all j modulo 12=5 sub-pixels etc. (See line charge control paragraph) . 
Between each "fall", the group of output charge buckets responsible for "catching" that 
sub-pixel charge is moved under its column thereby adding its charge contribution to the 
appropriate convolution value. When all sub-pixel charges for a given convolution value 
have been collected, the output bucket pair is read-out using double correlated double 
sampling as is done for the TMFPC. This read-out action frees the buckets for beginning 
another convolution elsewhere in the image. It is important to note that, unlike the 
TMFPC, the SMFPC active area is identical in structure to normal imaging CCD structures. 

3.1 Device Geometry Considerations 

The number of sub-pixels required of a convolution depends upon the sum of the abso- 
lute integer coefficients in the kernel. This number can be any integer value, not always 
a perfect square like 4, 9, 16 etc. Thus, it is appropriate to discuss logical pixel 
geometries that are other than square but which form some uniform grid. The repetition of 
a logical pixel shape can be described by a pair of "lattice" basis translations (a, b) 
where each vector a, b has integer coordinates. If a = (a x , ^ ) and b = (b x , b ), then the 
area of the unit cell is |a x b| = |a x b y - a y b x | which determines the number of sub-pixels 
contained therein. Figure 6 illustrates how logical pixel geometries can be formed for 
specific types of integer Kernels. The figure illustrates how a geometry of 12 sub-pixels 
for the I xx and I yy convolutions can be generated. We first note that the desired convolu- 
tion Kernel is 



1-2 1 
1-2 1 
1-2 1 



and has the sum of 



its absolute integer entries equal to 12. We then select the smallest integers a a b b 
so that a x b y - a y b x =12. This is achieved by setting a = (3, -2) and b = (3* 2) which 
results xn the geometry of Figure 6. 

3.2 Line Charge Controj 

To facilitate the multiplexing of sub-pixel charges from each line into their appro- 
priate output buckets,, it is necessary to control the release of charge from each line in 
some manner so that the various output buckets can move into position (sequentially) to 
"catch" the charges from the line. In the design, a modulo 12 control structure is used as 
shown m Figure 7, It consists of three portions; a modulo 3, modulo 4, and modulo 2 
control. To accomplish this control, phases 2 and 4 have separate control gates over 
specific columns of the image. The modulo 3 control consists of two such 4 phase struc- 
tures for which the first controls the passage of charge through column 1 and through 
columns (2, 3) while the second structure controls the flow of charge through column (1, 2) 
and through column 3 . The structure can therefore implement delay in the flow of charge 
modulo 3. Modulo 4 and modulo 2 control work in a similar way. 

4. RELATED TECHNOLOGIES and EXPERIENCES 

There are a number of related technologies associated with the two Focal Plane 
Convolvers that should be described as background material for their full understanding. 
In this section we discuss: 



o Double Correlated Double Sampling 

o CCD Charge Transport and Charge Transport Efficiency 

o Equal Variance Mapping 

o Local Rotational Invariance 

o Uniformity Concerns 

o Other Devices on the FPC Reticle 
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Figure 3. Photograph of the Space Multiplexing Focal Plane Convolver Figure 4. Illustration of the SMFPC Concept 
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Figure 5. Stehlikgram of the Output Multiplexer 
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All {^Accumulated in Same Output Bucket, P. 

Figure t Sj.-.. M.ltiplexing Geometry for the I„ (or I ) Convolution 
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4.1 Double Correlated Double Sampling (DCDS) 

In our FPC application, we are using DCDS it to provide a voltage proportional to the 
difference between charge packets. Q. is first applied to the CCD eletrometer which pro- 
vides a signal to the serial input while * p is' closed shorting the Capacitor C to ground. 
* is then opened and the serial input charge is reset . This provides a negative voltage 
proportional to (-Q. ) to the operational amplifier 2. Then, the "positive" charge packet, 
Q + , is added to the serial input which provides a voltage proportional to (Q + -Q. ) at 
amplifier 2 which is then sampled by * s . Both input and $ p are then reset to make ready 
for the next packet pair. 



4.2 CCD Charge Transport and Charge Transfer Efficiency 

A charge-coupled device is an array of closely spaced capacitors. A three-phase n- 
channel (electron transport) CCD is shown in Figure 9 and herein described. (Four phase 
devices, however, are more common.) The six MOS capacitors or electrodes connected to e Y , 
o, , and 0 3 clock lines form the main body of the CCD while the input diode (ID) , the input 
gate (IG) , the output diode (OD) , and the output gate (OG) are the input and output struc- 
ture that injects and detects charge packets to and from the main CCD body. 

The operation of the device can be explained with the aid of Figure 10, which shows 
the various clock wave forms and the potential and charge distribution inside the device. 
As can be seen, charge is transported from capacitor to capacitor by applying the correct 
sequence of attractive and repulsive signals on the CCD gates. 

4.3 CCD Charge Transport and Charge Transport Efficiency 

In the process of transporting charge in a CCD, some of the charge might be held back. 

Charge transfer efficiency, E, is one (1) minus the percent holdback over a 3 (or 4) gate 
transfer. 

In a CCD, the rate at which charge is transferred depends upon several factors: 

o Transverse fringe field if any" 

o Election diffusion 

o Self generated transverse fields. 

In the usual case, the gates are so long relative to the channel depth that the expo- 
nentially varying fringe fields die out too fast before they can influence the migration of 
charge into the accepting cell. One normally relies on self generated transverse fields 
and election diffusion currents. A simple theory shows that the last bit of charge to be 
transferred is controlled by diffusion and that the time constant, t, in seconds is propor- 
tional to the square of the gate length L. 



A transverse field is required to achieve faster transfer rates. Two techniques for 
building a transverse field inco the design are: 

o Peristaltic CCD's 
o Triangular gates 

The peristaltic approach involves a deeply burried channel where the bottom of the 
channel is roughly .4 times the gate length. At .5^ depth, the gate lengths should be 
1.25^! These are VHSIC design rules which could severely affect yield for CCD designs. An 
alternative approach is to use triangular gates in conjunction with deeply burried chan- 
nels. Here, the fringe fields cooperate to generate transverse fields and the gate geome- 
tries are not as severe. Such a triangular gate linear CCD array is on the FPC reticle. 
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Figure 8. Double Correlated Double Sampling Circuit 



t = .046 L 2 when L is measured in cm. 
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Upon testing this device, we achieved nearly .99999 charge transfer efficiency demon- 
strating that the approach could be used for high speed designs. The triangular gate 
concept is used in the SMFPC design. 
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Figure 9. A Three-Phase, N-Channel Charge-Coupled 
Device Cross-Sectional 
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Figure 11. Four Phase CCD Device 



Figure 10. Sketch of Potential and Charge 

Distribution of the Device Shown 



4.4. Equal Variance Mapping 

For some sensors, in particular visible imagery, the measured intensity at a pixel 
site will have a noise component which depends upon the number of photo-converted electrons 
present. This intensity dependent noise is due to the random nature of the photon emission 
process and is called shot noise. Prior to spacial processing, consideration was given to 
remapping the intensity space to an equal variance space; a square root-like mapping 
function is desired prior to convolutions. The TMFPC provides a charge partition gate 
within a resistive gate region for this purpose. The charge partitioning option provides 
the desired mapping function on the TMFPC. 



4. a. local Rotational Invariance 



Gradient Magnitudes, Gaussian Curvature, Laplacian and other "contractions" of tensor 
quantities are scalars. Scalars are rotational invariances. They are measures of the 
object being sensed and are independent of the coordinate system orientation. It can be 
observed that many of these non-linear scalars can be written as the sum or difference of 
the squares of linear convolution operators . For example, (1) the square gradient (edge 
detector) is 

G 2 = I 2 * + I 2 y/ (2) the Gaussian Curvature (corner detector) is C = 4 (I x T - I 2 ) 
= (l xx + I yy ) 2 - (I xx - I ry ) 2 - (2I xy ) 2 and (3) the line end operator 
^ " 2 I lxy (I„ y - I ) + i xyy - i jxi ) 

= - 1/2 I yyx ) 2 - (I xxx - 3/2 I yyx ) 2 + (1,^ - 1/2 I Ixy ) 2 - (I yyy - 3/2 I xxy ) 2 

For these reasons, analog squaring circuits have been included as test devices on the 
FPC reticle. 



4 . 6 Uniformity Concerns 



To achieve reasonable convolution processing accuracy e.g (6 to 8 bits) the charges 
in neighboring pixels must properly represent their photon flux. Two sources of non-uni- 
formity are: 

o Quantum efficiency variations 
o Non-uniform leakage 

Pixel to pixel variations in quantum efficiency will show up as false spacial patterns 
in regions of high optical intensity. This non-uniformity can come about because of trans- 
parency variations in the Si0 2 insulator or in the active pixel resistive gate region. It 
could also come about as a consequence of area non-uniformity. By using thinner dielec- 
trics and Sn0 2 transparent electrodes, the fluctuations due to material variations at pixel 
sites can be reduced to tolerable levels. 
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Another source of pixel non-uniformity is leakage. This leakage is a thermal charge 
generation effect which is significantly reduced at low e.g. (77°K) operational tempera- 
ture. For room temperature operations, leakage will lead to false spacial patterns. 
Fortunately, these patterns are independent of the input image and leads to background 
convolutions which can be subtracted from the convolved imagery. 

4.7 Other Devices on the FPC Reticle 

The following is a list of devices that were fabricated with the TMFPC and SMFPC 
devices. Space does not permit us to describe these devices or their test status: 

4.8 FPC Device 

o 11x11 Time Multiplexing Focal Plane Convolver 

o 11x11 Time Multiplexing Focal Plane Convolver 

o 120 x 120 Space Multiplexing Focal Plane Convolver 

4.9 Test Devices 

o Time-Mux center cell 
o Line arrays 
o Charge replication cell 
o CCD squaring circuit 
o Electrometer/Amplif ier 

o 12-channel SMFPC control gate test structure 
o 12 channel output multiplexing shift register 

In addition, there are a large variety of Process Test Structures . Process test 
devices exist for performing all the standard wafer processing tests and process control 
tests using. 

5. Summary and Conclusions 

Time multiplexing and space multiplexing focal plane convolvers useful for target 
detection and tracking were designed, fabricated and tested. Three wafer lots of twenty 
wafers each were processed. As a consequence of testing on the devices of the first two 
lots, all test anomalies were understood and steps were taken to correct the mask errors 
and modify the process recipe. Upon testing of some third lot devices, it was found that 
the modified process recipe did indeed lead to much better uniformity, charge handling 
capacity and greater charge transfer efficiency. However, there are still two problems: 

o High resistivity intra-layer short between two TMFPC gates 
o Inter-layer shorts between the two polysilicon and between 
Polysilicon and substrate. 

Program funding did not permit us to perform the QA laboratory wafer tests needed to 
isolate the location of the intra-layer shorts. 

Because all aspects of the TMFPC array have been made to work, we are assured of a 
working TMFPC should the two existing deficiencies be corrected. 
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